1. Field of the Invention
This invention relates to an image processing method and apparatus. This invention particularly relates to an image processing method and apparatus, wherein a specific image portion, such as an abnormal pattern or a high-contrast image portion, which is embedded in an image, is emphasized selectively.
2. Description of the Prior Art
Image processing, such as gradation processing or frequency processing, has heretofore been carried out on an image signal, which represents an image and has been obtained with one of various image obtaining methods, such that a visible image having good image quality can be reproduced and used as an effective tool in, particularly, the accurate and efficient diagnosis of an illness. Particularly, in the field of medical images, such as radiation images of human bodies serving as objects, it is necessary for specialists, such as doctors, to make an accurate diagnosis of an illness or an injury of the patient in accordance with the obtained image. Therefore, it is essential to carry out the image processing in order that a visible image having good image quality can be reproduced and used as an effective tool in the accurate and efficient diagnosis of an illness.
As one of the image processing, frequency emphasis processing has been disclosed in, for example, Japanese Unexamined Patent Publication No. 61(1986)-169971. With the disclosed frequency emphasis processing, an image signal (i.e., an original image signal) Dorg representing the image density value of an original image is converted into an image signal Dproc with Formula (36). EQU Dproc=Dorg+.beta..times.(Dorg-Dus) (36)
wherein .beta. represents the frequency emphasis coefficient, and Dus represents the unsharp mask signal. The unsharp mask signal Dus comprises super-low frequency components obtained by setting a mask, i.e. an unsharp mask, constituted of a picture element matrix, which has a size of N columns .times.N rows (wherein N represents an odd number) and has its center at the picture element represented by the original image signal Dorg, in a two-dimensional array of picture elements. The unsharp mask signal Dus is calculated with, for example, Formula (37). EQU Dus=(.SIGMA.Dorg)/N.sup.2 (37)
wherein .SIGMA.Dorg represents the sum of the image signal values representing the picture elements located within the unsharp mask.
The value of (Dorg-Dus) in the parenthesis of the second term of Formula (36) is obtained by subtracting the unsharp mask signal, which represents the super-low frequency components, from the original image signal. Therefore, comparatively high frequency components can be extracted selectively by subtracting the super-low frequency components from the original image signal. The comparatively high frequency components are then multiplied by the frequency emphasis coefficient .beta., and the obtained product is added to the original image signal. In this manner, the comparatively high frequency components can be emphasized.
Also, iris filter processing (hereinbelow often referred to as the operation of the iris filter) has heretofore been known as the operation processing for selectively extracting only a specific image portion, such as an abnormal pattern, from an image. [Reference should be made to "Detection of Tumor Patterns in DR Images (Iris Filter)," Obata, et al., Collected Papers of The Institute of Electronics and Communication Engineers of Japan, D-II, Vol. J75-D-II, No. 3, pp. 663-670, March 1992.] The iris filter processing has been studied as a technique efficient for detecting, particularly, a tumor pattern, which is one of characteristic forms of mammary cancers. However, the image to be processed with the iris filter is not limited to the tumor pattern in a mammogram, and the iris filter processing is applicable to any kind of image having the characteristics such that the gradients of the image signal representing the image are centralized.
How the processing for detecting the image portion with the iris filter is carried out will be described hereinbelow by taking the processing for the detection of the tumor pattern as an example.
It has been known that, for example, in a radiation image recorded on a negative X-ray film (i.e., an image yielding an image signal of a high signal level for a high image density), the density values of a tumor pattern are slightly smaller than the density values of the surrounding image areas. The density values of the tumor pattern are distributed such that the density value becomes smaller from the periphery of an approximately circular tumor pattern toward the center point of the tumor pattern. Therefore, in the tumor pattern, gradients of the density values can be found in local areas, and the gradient lines (i.e., gradient vectors) centralize in the directions heading toward the center point of the tumor pattern.
With the iris filter, the gradients of image signal values, which are represented by the density values, are calculated as gradient vectors, the degree of centralization of the gradient vectors is calculated, and a tumor pattern is detected in accordance with the calculated degree of centralization of the gradient vectors. Specifically, the gradient vector at an arbitrary picture element in a tumor pattern is directed to the vicinity of the center point of the tumor pattern. On the other hand, in an elongated pattern, such as a blood vessel pattern, gradient vectors do not centralize upon a specific point. Therefore, the distributions of the directions of the gradient vectors in local areas may be evaluated, and a region, in which the gradient vectors centralize upon a specific point, may be detected. The thus detected region may be taken as a prospective tumor pattern, which is considered as being a tumor pattern. The processing with the iris filter is based on such fundamental concept. Steps of algorithms of the iris filter will be described hereinbelow.
(Step 1) Calculation of gradient vectors
For each picture element j among all of the picture elements constituting a given image, the direction .theta. of the gradient vector of the image signal representing the image is calculated with Formula (38). ##EQU1##
As illustrated in FIG. 5, f.sub.1 through f.sub.16 in Formula (38) represent the density values (i.e., the image signal values) corresponding to the picture elements located at the peripheral areas of a mask, which has a size of five picture elements (located along the column direction of the picture element array) .times.five picture elements (located along the row direction of the picture element array) and which has its center at the picture element j.
(Step 2) Calculation of the degree of centralization of gradient vectors
Thereafter, for each picture element among all of the picture elements constituting the given image, the picture element is taken as a picture element of interest, and the degree of centralization C of the gradient vectors with respect to the picture element of interest is calculated with Formula (39). ##EQU2##
As illustrated in FIG. 6, in Formula (39), N represents the number of the picture elements located in the region inside of a circle, which has its center at the picture element of interest and has a radius R, and .theta.j represents the angle made between the straight line, which connects the picture element of interest and each picture element j located in the circle, and the gradient vector at the picture element j, which gradient vector has been calculated with Formula (38). Therefore, in cases where the directions of the gradient vectors of the respective picture elements j centralize upon the picture element of interest, the degree of centralization C represented by Formula (39) takes a large value.
The gradient vector of each picture element j, which is located in the vicinity of a tumor pattern, is directed approximately to the center portion of the tumor pattern regardless of the level of the contrast of the tumor pattern. Therefore, it can be regarded that the picture element of interest associated with the degree of centralization C, which takes a large value, is the picture element located at the center portion of the tumor pattern. On the other hand, in a linear pattern, such as a blood vessel pattern, the directions of the gradient vectors are biased to a certain direction, and therefore the value of the degree of centralization C is small. Accordingly, a tumor pattern can be detected by taking each of all picture elements, which constitute the image, as the picture element of interest, calculating the value of the degree of centralization C with respect to the picture element of interest, and rating whether the value of the degree of centralization C is or is not larger than a predetermined threshold value. Specifically, the processing with the iris filter has the features over an ordinary difference filter in that the processing with the iris filter is not apt to be adversely affected by blood vessel patterns, mammary gland patterns, or the like, and can efficiently detect tumor patterns.
In actual processing, such that the detection performance unaffected by the sizes and shapes of tumor patterns may be achieved, it is contrived to adaptively change the size and the shape of the filter. FIG. 7 shows an example of the filter. The filter is different from the filter shown in FIG. 6. With the filter of FIG. 7, the degree of centralization is rated only with the picture elements, which are located along radial lines extending radially from a picture element of interest in M kinds of directions at 2.pi./M degree intervals. (In FIG. 7, by way of example, 32 directions at 11.25 degree intervals are shown.)
In cases where the picture element of interest has the coordinates (k, 1), the coordinates ([x], [y]) of the picture element, which is located along an i'th radial line and is the n'th picture element as counted from the picture element of interest, are given by Formulas (40) and (41). EQU x=k+n cos {2.pi.(i-1)/M} (40) EQU y=l-n sin {2.pi.(i-1)/M} (41)
wherein [x] represents the maximum integer, which does not exceed x, and [y] represents the maximum integer, which does not exceed y.
Also, for each of the radial lines, the output value obtained for the picture elements ranging from the picture element of interest to a picture element, which is located along the radial line and at which the maximum degree of centralization is obtained, is taken as the degree of centralization with respect to the direction of the radial line. The mean value of the degrees of centralization, which have been obtained for all of the radial lines, is then calculated. The mean value of the degrees of centralization having thus been calculated is taken as the degree of centralization C of the gradient vector group with respect to the picture element of interest.
Specifically, the degree of centralization Ci(n), which is obtained for the picture elements ranging from the picture element of interest to the n'th picture element located along the i'th radial line, is calculated with Formula (42). ##EQU3## wherein Rmin and Rmax respectively represent the minimum value and the maximum value having been set for the radius of the tumor pattern, which is to be detected.
The calculation of the degree of centralization Ci(n) may be carried out by using Formula (42') in lieu of Formula (42). ##EQU4##
Specifically, with Formula (42'), the degree of centralization Ci(n) is obtained for the picture elements, which are located along the i'th radial line and fall within the range from an Rmin'th picture element, that corresponds to the minimum value Rmin, as counted from the picture element of interest, to an n'th picture element, that falls within the range from the Rmin'th picture element to an Rmax'th picture element corresponding to the maximum value Rmax, as counted from the picture element of interest.
Thereafter, the degree of centralization C of the gradient vector group is calculated with Formulas (43) and (44). ##EQU5##
Formula (43) represents the maximum value of the degree of centralization Ci(n) obtained for each of the radial lines with Formula (42) or (42'). Therefore, the region from the picture element of interest to the picture element associated with the degree of centralization Ci(n), which takes the maximum value, may be considered as being the region of the prospective tumor pattern. By the detection of such regions for all of the radial lines with Formula (43), it is possible to judge the shape of the peripheral edge of the region, which may be regarded as the prospective tumor pattern.
With Formula (43), the maximum values of the degrees of centralization within the aforesaid regions are calculated for all directions of the radial lines. Thereafter, with Formula (44), the mean value of the maximum values of the degrees of centralization within the aforesaid regions, which maximum values have been given by Formula (43) for all directions of the radial lines, is calculated. The calculated mean value is compared with a predetermined threshold value T. From the results of the comparison, a judgment is made as to whether there is or is not a probability that the region having its center at the picture element of interest will be the abnormal pattern.
The region, in which the degree of centralization C of the gradient vector group with Formula (44) is rated, is similar to the iris of the human's eye, which expands or contracts in accordance with the brightness of the external field. The size and the shape of the region is changed adaptively in accordance with the distribution of the gradient vectors. Therefore, the filter used is referred to as the iris filter.
(Step 3) Rating of the shape and form of the tumor pattern
In general, patterns of malignant tumors have the characteristics of the shapes and forms described below.
1) The side edges are irregular.
2) The shape is close to an ellipse.
3) The region inside of the pattern has a convex or concave density distribution.
Therefore, a judgment is made as to the shape and form by considering these characteristics such that patterns of normal tissues may be eliminated from the detected prospective pattern, and such that only the pattern considered as being a tumor pattern, can be detected. The characteristic measures used in making the judgment include the spreadness, the elongation, the roughness of side edges, the circularity, and the degree of convexity or concavity (i.e., the entropy) of the density distribution in the region inside of the pattern.
For example, the circularity may be employed as the characteristic measure for the shape judgment. In such cases, when the degrees of centralization are binarized, the distribution of the binarized degrees of centralization corresponding to the tumor pattern ordinarily takes a shape close to a circle. The diameter of the circle having the same area as the area of the region obtained from the binary conversion is represented by Le. Also, the lengths of the longitudinal side and the lateral side of a square, which has the minimum area capable of accommodating the region, are respectively represented by a and b. In such cases, the circularity d.sub.circ is defined by Formula (45). EQU d.sub.circ =Le/)a+b) wherein Le=2(S/.pi.).sup.1/2 (45)
In cases where the value of the circularity is smaller than a predetermined threshold value, it is judged that the region is not a tumor pattern, and the region is not detected as the tumor pattern. In cases where the value of the circularity is not smaller than the predetermined threshold value, it is judged that the region is a tumor pattern, and the region is detected as the tumor pattern.
By carrying out the steps described above, the iris filter can efficiently detect a tumor pattern from a radiation image.
Further, processing based upon the algorithm of morphology (hereinbelow referred to as the morphology operation or the morphology processing) has heretofore been known as the operation processing for selectively extracting only a specific image portion, such as an abnormal pattern, from an image.
The morphology processing has been studied as a technique efficient for detecting, particularly, a small calcified pattern, which is one of characteristic forms of mammary cancers. However, the image to be processed with the morphology processing is not limited to the small calcified pattern in a mammogram, and the morphology processing is applicable to any kind of image, in which the size and the shape of a specific image portion (i.e., an abnormal pattern, or the like) to be detected are known previously.
The morphology processing is carried out by using a multi-scale .lambda. and a structure element (i.e., a mask) B. The morphology processing has the features in that, for example, (1) it is efficient for extracting a calcified pattern itself, (2) it is not affected by complicated background information, and (3) the extracted calcified pattern does not become distorted.
Specifically, the morphology processing is advantageous over ordinary differentiation processing in that it can more accurately detect the geometrical information concerning the size, the shape, and the density distribution of the calcified pattern. How the morphology processing is carried out will be described hereinbelow by taking the detection of a small calcified pattern in a mammogram as an example.
(Fundamental operation of morphology processing)
In general, the morphology processing is expanded as the theory of sets in an N-dimensional space. As an aid in facilitating the intuitive understanding, the morphology processing will be described hereinbelow with reference to a two-dimensional gray level image.
The gray level image is considered as a space, in which a point having coordinates (x, y) has a height corresponding to a density value f(x, y). In this case, it is assumed that the image signal representing the density value f(x, y) is a high luminance-high signal level type of image signal, in which a low density (i.e., a high luminance when the image is displayed on a CRT display device) is represented by a high image signal level.
Firstly, as an aid in facilitating the explanation, a one-dimensional function f(x) corresponding to the cross section of the two-dimensional gray level image is considered. It is assumed that structure element g used in the morphology operation is a symmetric function of Formula (46), which is symmetric with respect to the origin. EQU g.sup.s (X)=g(-X) (46)
It is also assumed that the value is 0 in a domain of definition G, which is represented by Formula (47). EQU G={-m, -m+1, . . . , -1, 0, 1, . . . , m-1, m} (47)
In such cases, the fundamental forms of the morphology operation are very simple operations carried out with Formulas (48), (49), (50), and (51). EQU dilation; [f.sym.G.sup.s ](i)=max {f(i-m), . . . , f(i), . . . , f(i+m)}(48) EQU erosion; [f .crclbar.GS] (i)=min {f(i-m), . . . , f(i), . . . , f(i+m)}(49) EQU opening; fg=(f.crclbar.g.sup.s).sym.g (50) EQU closing; fg=(f.sym.g.sup.s).crclbar.g (51)
Specifically, as illustrated in FIG. 14A, the dilation processing is the processing for retrieving the maximum value in the range of a width of .+-.m (which is the value determined in accordance with a structure element B and corresponding to the mask size shown in FIG. 14A) having its center at a picture element of interest. As illustrated in FIG. 14B, the erosion processing is the processing for retrieving the minimum value in the range of the width of .+-.m having its center at the picture element of interest.
The signal (indicated by the broken line in FIG. 14A), which is obtained from the dilation processing, or the signal (indicated by the broken line in FIG. 14B), which is obtained from the erosion processing, may then be subtracted from the original image signal. In this manner, as indicated by the portions hatched in FIG. 14A or FIG. 14B, it is possible to obtain a morphology signal Dmor having values other than zero only for an image edge portion, at which the signal value changes sharply, and an image portion, at which the signal values fluctuate within a range spatially smaller than the structure element.
The opening processing is equivalent to the processing in which the dilation processing is carried out after the erosion processing, i.e., the processing in which the maximum value is searched after the searching of the minimum value. Also, the closing processing is equivalent to the processing in which the erosion processing is carried out after the dilation processing, i.e., the processing in which the minimum value is searched after the searching of the maximum value.
More specifically, as illustrated in FIG. 14C, the opening processing is equivalent to the processing for smoothing the density curve f(x) from the low luminance side, and removing a convex density fluctuating portion (i.e., the portion at which the luminance is higher than that of the surrounding portions), which fluctuates in a range spatially narrower than the mask size of 2 m. Also, as illustrated in FIG. 14D, the closing processing is equivalent to the processing for smoothing the density curve f(x) from the high luminance side, and removing a concave density fluctuating portion (i.e., the portion at which the luminance is lower than that of the surrounding portions), which fluctuates in the range spatially narrower than the mask size of 2 m.
The signal (indicated by the broken line in FIG. 14C), which is obtained from the opening processing, or the signal (indicated by the broken line in FIG. 14D), which is obtained from the closing processing, may then be subtracted from the original image signal. In this manner, as indicated by the portions hatched in FIG. 14C or FIG. 14D, it is possible to obtain a morphology signal Dmor having values other than zero only for a characteristic image portion, at which the signal values fluctuate within a range spatially smaller than the structure element.
In cases where the structure element g is not symmetric with respect to the origin, the dilation operation with Formula (48) is referred to as the Minkowski sum, and the erosion operation with Formula (49) is referred to as the Minkowski difference.
In cases where the image signal representing the density value f(x) is a high density-high signal level type of image signal, in which a high density is represented by a high image signal level, the relationship between the density value f(x) and the image signal value becomes reverse to the relationship between the density value f(x) and the image signal value in the high luminance-high image signal level type of image signal. Therefore, the dilation processing, which is carried out on the high density-high signal level type of image signal, coincides with the erosion processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 14B. The erosion processing, which is carried out on the high density-high signal level type of image signal, coincides with the dilation processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 14A. The opening processing, which is carried out on the high density-high signal level type of image signal, coincides with the closing processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 14D. Also, the closing processing, which is carried out on the high density-high signal level type of image signal, coincides with the opening processing, which is carried out on the high luminance-high signal level type of image signal as shown in FIG. 14C.
The morphology processing is herein described with respect to the high luminance-high signal level type of image signal.
(Application to detection of calcified patterns)
In order for a calcified pattern to be detected, it is considered to employ a difference method, in which a smoothed image signal is subtracted from the original image signal. However, with a simple smoothing method, it is difficult to discriminate the calcified pattern from an elongated non-calcified pattern (for example, a pattern of the mammary gland, a blood vessel, mammary gland supporting tissues, or the like). Therefore, Obata of Tokyo University of Agriculture and Technology, et al. have proposed a morphology filter, which is represented by Formula (52) and is based upon the opening operation using a multiply structure element. [Reference should be made to "Extraction of Small Calcified Patterns with A Morphology Filter Using A Multiply Structure Element," Collected Papers of The Institute of Electronics and Communication Engineers of Japan, D-II, Vol. J75-D-II, No. 7, pp. 1170-1176, July 1992; and "Fundamentals of Morphology and Its Application to Mammogram Processing," Medical Imaging Technology, Vol. 12, No. 1, January 1994.] ##EQU6##
In Formula (52), Bi (wherein i=1, 2, . . . , M) represents M number of linear structure elements (M=4 in the example shown in FIG. 12). (The M number of structure elements, as a whole, will hereinbelow be referred to as the multiply structure element. Also, the multiply structure element will often be simply referred to as the structure element, including the cases where i=1.) In cases where the structure element Bi is set to be larger than the calcified pattern to be detected, a calcified pattern, which is a convex signal change portion finer than the structure element Bi (i.e., which is an image portion fluctuating in a spatially narrow range) and has luminance values larger than the luminance values of the surrounding portions, is removed in the opening processing. On the other hand, an elongated non-calcified pattern is longer than the structure element Bi. Therefore, in cases where the inclination of the non-calcified pattern (i.e, the direction along which the non-calcified pattern extends) coincides with one of the directions of the four structure elements Bi, the non-calcified pattern remains unremoved after the opening processing, i.e. the operation of the second term of Formula (52), has been carried out. Therefore, when the smoothed image signal obtained from the opening processing (i.e. the signal representing the image, from which only the calcified pattern has been removed) is subtracted from the original image signal f, an image can be obtained which contains only the small calcified pattern. This is the concept behind Formula (52).
As described above, in cases where the image signal is of the high density-high signal level type, the density value of the calcified pattern is smaller than the density values of the surrounding image portions, and the calcified pattern constitutes a concave signal change portion with respect to the surrounding portions. Therefore, the closing processing is applied in lieu of the opening processing, and Formula (53) is applied in lieu of Formula (52). ##EQU7##
As described above, in order that a visible image having good image quality can be reproduced and used as an effective tool in, particularly, the accurate and efficient diagnosis of an illness, it is essential to carry out the image processing on the given image. However, in cases where the image processing is carried out for the entire area of the image as in the conventional techniques or in cases where the emphasis processing merely depending on the image density is carried out as disclosed in, for example, Japanese Unexamined Patent Publication No. 2(1990)-1078, components adversely affecting the image quality, such as radiation noise components in a mammogram, are also emphasized. As a result, the image quality of the image and its capability of serving as an effective tool in, particularly, the efficient and accurate diagnosis of an illness become low.
Also, as disclosed in, for example, Japanese Patent Publication No. 60(1985)-192482, Japanese Unexamined Patent Publication No. 2(1990)-120985, and Japanese Patent Application Publication No. 3(1991)-502975, in cases where emphasis processing depending upon the value of variance of an image signal is carried out, an image portion having a locally large change in density is emphasized to a high extent. Therefore, the problems occur in that undershooting and overshooting become relatively perceptible in the vicinity of the image portion. Particularly, as for X-ray images, an artifact is apt to occur on the high density side.